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Preface 


I  chose  the  area  of  critical  thinking  for  the  focus  of  this  research  project  because  of 
my  interest  in  education  and  my  behef  that  innovative  problem  solving,  critical  analytical 
thought  and  sound  professional  judgment  have  been  and  will  continue  to  be  keys  to 
battlespace  dominance.  Additionally,  since  ACSC  focuses  on  shaping  and  molding 
tomorrow’s  leaders,  this  research  can  contribute  to  the  continued  improvement  of  the 
professional  mihtary  education  it  provides.  While  much  progress  has  been  made 
transforming  the  ACSC  curriculum  from  a  passive  one,  based  primarily  on  lectures,  to  an 
active  seminar  based  discussion  format,  I  beheve  that  the  area  of  assessment  needs 
additional  emphasis  to  vahdate  that  these  new  improvements  are  in  fact  improving 
students’  abihties  to  think  critically.  My  recommendations  provide  areas  that  need  further 
study  and  I  hope  to  conduct  further  research  in  support  of  a  dissertation  to  complete  my 
Doctorate  in  Education. 

I  am  greatly  indebted  to  Maj  Brenda  Roth  (PhD)  for  her  guidance  and  assistance. 
Her  dissertation  on  student  outcomes  assessment  at  ACSC  provided  a  foundation  for  my 
research  and  allowed  me  to  focus  on  critical  thinking,  a  specific  area  of  her  work.  She 
was  an  invaluable  research  advisor  giving  both  encouragement  and  constructive  criticism. 
Lt  Col  Michael  Conn  also  was  quite  helpful  and  provided  information  on  Myers-Briggs 
Type  Indicator  research  and  feedback  on  the  overall  ACSC  evaluation  system. 
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Abstract 

Critical  thinking  is  presently  an  area  of  emphasis  at  the  Air  Force’s  Air  Command  and 
Staff  College  (ACSC).  This  project  provides  the  background  information  and  framework 
to  assess  ACSC’s  success  at  fulfiUing  its  goal  of  fostering  critical  thinking.  Along  the  way 
it  answers  four  sahent  questions  concerning  critical  thinking.  First,  why  critical  thinking  is 
so  vitally  important  for  leaders  today.  The  world’s  rapidly  changing  contextual  factors 
and  turbulent  environment  dictates  that  critical  thinking  be  a  necessary  characteristic  in 
leadership  development.  Secondly,  the  problem  with  defining  exactly  what  is  meant  by  the 
term  critical  thinking;  The  lack  of  a  consensus  on  a  precise  definition  is  overcome  by 
synthesizing  a  definition  that  contributes  to  the  solution  of  the  two  remaining  questions: 
how  to  assess  critical  thinking  and  how  is  critical  thinking  fostered?  A  number  of  different 
evaluation  techniques  are  analyzed  with  the  conclusion  that  the  Watson-Glaser  Appraisal 
combines  the  best  combination  of  vahdity,  rehabihty  and  ease  of  use.  Four  instructional 
techniques  are  also  evaluated  with  current  research  hterature  suggesting  a  more  exphcit 
teaching  approach  (infusion)  might  be  beneficial  for  ACSC.  A  critical  analysis  of  the 
current  program  indicates  that  although  critical  thinking  is  being  fostered,  effectiveness 
and  efficiency  cannot  be  determined  with  the  descriptive  evaluation  data  presently 
available;  this  deficiency  could  be  eliminated  with  the  use  of  the  Watson-Glaser  appraisal. 
Areas  for  additional  research  are  also  suggested. 
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Chapter  1 


Introduction 

The  dynamic  nature  of  joint  operations  in  the  21st  century  battlespace  will 
require  a  continued  emphasis  on  developing  strong  leadership 
skills....  Effective  leadership  provides  our  greatest  hedge  against 
uncertainty. 

—  Joint  Vision  2010 

Background  and  Problem  Definition 

Leadership  is  important  in  any  organization  and  this  is  especially  true  in  the  mihtary/ 
Researchers  have  proposed  many  leadership  models  over  the  years  in  an  attempt  to 
develop  theories  that  describe,  predict,  evaluate,  and  develop  better  leaders.  These  efforts 
to  develop  leadership  theories  have  been  only  marginally  successful  and  thus  new  and 
more  complex  theories  have  continued  to  evolve  in  an  attempt  to  fiU  this  void.  The 
premise  of  this  research  paper  is  that  the  world’s  rapidly  changing  contextual  factors  and 
the  increased  importance  of  interdependence  force  us  to  look  at  leadership  in  a 
revolutionary  new  way — leadership  as  a  process  rather  than  a  position.^  Furthermore, 
because  critical  thinking  is  a  primary  tool  for  dealing  with  the  many  dilemmas  and 
paradoxes  in  today’s  turbulent  environment,  the  support  of  critical  thinking  is  a  necessary 
characteristic  in  effective  leadership  development.^ 
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Basic  Air  Force  doctrine  acknowledges  this  connection  between  leadership  and 
critical  thinking  and  provides  the  following  general  guidance:  “Professional  mihtary 
education  should  encourage  critical  analytical  thought,  innovative  problem  solving  and 
sound  professional  judgment.”'^  Based  on  this  guidance  the  Air  Command  and  Staff 
College  has  incorporated  “Preparing  leaders  to  think  strategically,  operationally,  and 
critically”  as  one  of  its  stated  goals. ^  However,  problems  arise  when  one  begins  to  look 
closer  at  the  specifics:  Exactly  what  is  critical  thinking?  How  can  it  be  measured?  What  is 
the  best  way  to  foster  it?  Finally  culminating  with  the  bottom  hue  question:  Is  ACSC 
succeeding  in  fulfilling  its  critical  thinking  goal? 

Methodology 

In  order  to  answer  these  questions,  a  hterature  search  was  conducted  to  identify  and 
obtain  as  much  current  information  on  the  area  of  critical  thinking  as  possible.  Recent 
empirical  studies  were  primarily  sought;  however,  older  studies  were  also  used  if  they 
were  judged  to  add  value.  The  references  used  in  this  paper  were  obtained  by  electronic 
searches  of  the  ERIC  database  (1986-1996)  and  the  on- hue  catalog  at  the  University  of 
Southern  Cahfornia  and  Air  University  Eibraries.  Hand  searches  were  also  conducted  for 
the  past  year  of  the  following  journals:  American  Educational  Research  Journal, 
Cognition  and  Instruction,  Educational  Psychologist,  Higher  Education,  Journal  of 
Educational  Psychology,  and  the  Review  of  Educational  Research.  Descriptors  used 
included  key  terms  such  as  leadership,  leadership  training,  professional  mihtary  education, 
critical  thinking,  metacognition,  and  cognitive  strategies  along  with  a  number  of  authors 
who  speciahze  in  this  field  such  as  Robert  Ennis,  Stephen  Norris,  Harvey  Siegel,  and 
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Richard  Paul.  Finally  a  thorough  search  of  the  Internet  World  Wide  Web  also  uncovered 
several  web  sites,  for  example,  (http://www.sonoma.edu/cthink/)  dedicated  to  this  area  of 
study  that  yielded  a  considerable  amount  of  information.  Additional  information  was  also 
obtained  from  personal  interviews  with  ACSC  leadership  and  staff  along  with  a  review  of 
curriculum,  teaching  methods,  and  evaluation  techniques. 

Overview 

This  project  will  provide  the  background  and  a  framework  that  can  be  used  to  assess 
Air  Command  and  Staff  College  curriculum,  instructional  techniques,  evaluations,  and 
staff  development  in  relation  to  its  goal  of  fostering  critical  thinking  and  this  current 
research.  Along  the  way  it  will  answer  four  sahent  questions  concerning  critical  thinking. 
First,  the  problem  with  defining  exactly  what  is  meant  by  the  term  critical  thinking  ;  The 
lack  of  a  consensus  on  a  precise  definition  contributes  to  the  problems  discussed  in  the 
next  two  questions:  how  to  assess  critical  thinking  is  the  second  area  of  discussion?  A 
number  of  different  evaluation  techniques  presently  used  in  research  will  be  analyzed 
according  to  their  strengths  and  weaknesses  in  an  attempt  to  select  one  that  might  be  used 
in  the  future  for  ACSC.  The  preceding  support  the  discussions  to  the  third  question:  how 
can  critical  thinking  best  be  fostered  through  instruction?  Four  instructional  techniques, 
infusion,  immersion,  general  and  mixed  approach  will  be  evaluated  on  which  might  be 
most  beneficial  for  ACSC  instruction.  Finally,  the  research  will  be  summarized  in  an 
attempt  to  assess  the  current  ACSC  program  to  foster  critical  thinking  and  proposals  will 
be  given  that  can  be  used  to  validate  and  improve  this  area  of  instruction. 
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Notes 


^  Joint  Vision  2010,  America’s  Military:  Preparing  For  Tomorrow 
^  Richard  L.  Hughes,  Robert  C.  Ginnett,  and  Gordon  J.  Curphy,  Leadership: 
Enhancing  the  Lessons  of  Experience  (Burr  Ridge,  Illinois:  R.  R.  Donnelley  &  Sons 
Company,  1993),  1-18. 

^  Luke  NoveUi,  Jr.,  and  Sylvester  Taylor,  “The  Context  for  Leadership  in  the  21st- 
Century  Organizations,”  American  Behavioral  Scientist  37,  no.  1  (September  1993):  139- 
147. 

Air  Force  Manual  (AFM)  1-1,  Basic  Aerospace  Doctrine  of  the  United  States  Air 
Force,  vol.  1,  March  1992,  19. 

^  “Air  Command  and  Staff  College,”  Air  University  Catalog,  1995-96,  n.p.;  on- hue, 
Internet,  6  February  1997,  available  from  http://www.au.af.mil/au/cat/acsc.html. 
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Chapter  2 


The  Language  of  Critical  Thinking 

In  a  world  of  shallow  values,  instant  gratification,  and  quick  fixes,  critical 
thinking  is  for  those  few  that  see  the  benefit  of  intellectual  traits, 
standards  and  abilities  that  enable  them  to  cut  through  the  propaganda, 
the  information  blitz,  and  make  sense  of  the  world. 

—  Richard  Paul 

Critical  Thinking:  How  to  Prepare  Students  For  a  Rapidly  Changing  World 

Demystifying  the  Concept:  What  is  Critical  Thinking 

Air  Force  doctrine  states  that  leaders  need  to  be  able  to  think  critically  and  this 
attribute  should  be  fostered  through  professional  mihtary  education.  This  sounds 
reasonable  and  thus  the  Air  Command  and  Staff  College  (ACSC)  has  fostering  critical 
thinking  as  one  of  its  stated  goals.  However,  the  term  “critical  thinking”  is  no  where 
defined  in  Air  Force  pubhcations,  no  guidance  is  given  on  how  to  accomphsh  this  goal, 
and  no  criteria  is  provided  so  that  progress  may  be  assessed.  One  might  assume  that  these 
answers  are  readily  provided  by  academia,  and  therefore  they  do  not  need  to  be 
specifically  addressed.  However,  as  we  shall  see,  this  would  be  a  serious  mistake. 

Critical  thinking  scholarship  is  in  a  mystified  state.  There  is  no  single  definition  that 
is  widely  accepted  and  the  use  of  many  other  strange  terms  such  as  infusion,  immersion, 
general  and  mixed  approach,  just  add  to  the  confusion.^  To  rectify  this  situation,  we  need 
to  synthesize  a  definition  to  clarify  exactly  what  is  meant  by  the  term  critical  thinking. 
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Robert  Ennis,  one  of  the  leading  authors  in  eritical  thinking  from  the  University  of  Illinois 
at  Urbana-Champaign,  defines  eritieal  thinking  as  “reasonable  and  reflective  thinking  that 
is  focused  on  deciding  what  to  beheve  or  do.”^  John  McPeck,  another  leader  in  this  field 
and  a  critic  of  Ennis,  views  critical  thinking  as  “the  skill  and  propensity  to  engage  in  an 
activity  with  reflective  skepticism  within  the  context  of  a  discipline  and  the  knowledge 
within  a  given  field”^  Halonen  uses  these  definitions  from  the  two  extremes  of  the 
generahzabihty  debate  to  form  a  hybrid:  critical  thinking  is  the  propensity  and  skills  to 
engage  in  an  activity  with  reflective  skepticism  focused  on  deciding  what  to  reasonably 
believe  or  dof  Erom  this  it  can  be  seen  critical  thinking  has  (at  least)  two  central 
components.  The  first,  a  “reason  assessment”  component  involves  the  abihties  and  skills 
relevant  to  the  proper  understanding  and  evaluations  of  reasons,  claims,  and  arguments; 
The  second,  a  “critical  spirit”  component  that  is  characterized  by  a  propensity  to  apply 
their  critical  thinking  skills.  This  component  may  be  broken  down  into  a  complex  of 
dispositions,  attitudes,  habits,  and  traits.  Eor  instance,  open-mindedness  is  one  of  the 
more  important  “critical  spirit”  aspects.  Taking  another  look  at  the  definition  one  can  see 
it  hinges  on  three  key  words:  reasonable,  reflective,  and  focused. 

Criteria  for  Critical  Thinking 

The  first  criteria  is  that  critical  thinking  must  be  reasonable  as  opposed  to  arbitrary  or 
unreasonable.  It  must  rely  on  the  use  of  vahd  supporting  evidence  and  appropriate 
inference  from  which,  in  general,  the  best  conclusions  are  drawn.  Secondly,  critical 
thinkers  must  be  reflective.  They  must  consciously  evaluate  their  own  and  others’ 
thinking  in  an  effort  to  improve  it.  Third,  critical  thinking  is  focused  thinking.  It  is 
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thinking  with  a  purpose.  That  purpose  is  to  make  the  best  decision  about  what  to  beheve 
or  do.  Figure  1  provides  a  visual  representation  of  the  critical  thinking  process.^ 


Figure  1.  A  Pictorial  Representation  of  Critical  Thinking 

Looking  at  the  model,  we  can  see  that  it  depicts  the  central  criteria  (reasonable, 
reflective,  and  focused)  for  our  definition  of  critical  thinking.  The  entire  process  takes 
place  in  a  problem  solving  context,  this  focuses  the  thinking.  The  decision  rests  on  some 
basic  support  which  is  used  to  reasonably  infer  some  conclusion.  The  inference  link  is 
very  important  for  critical  thinking  to  proceed,  a  person  needs  both  the  skills  of  the 
“reason  assessment”  component  and  the  dispositions  of  the  “critical  spirit”  component. 
Reflection  acts  as  a  quahty  check  throughout  the  process.  Up  to  this  point  we  have  talked 
in  very  general  terms,  some  specific  examples  should  now  be  helpful.  Tables  1  and  2 


7 


provide  a  further  breakdown  of  the  reason  assessment  skills  and  disposition  components 


of  critical  thinking. 


Table  1.  Critical  Thinking  Reason  Assessment  Skills 


Elementary  Clarification 

Basic  Support 
Inference 

Advanced  Clarification 
Strategies  &  Tactics 

Source:  Stephen  P.  Norris  and  Robert  H.  ] 
Grove,  CA:  Critical  Thinking  Press 


Focusing  on  a  question 
Analyzing  arguments 

Asking/answering  questions  that  clarify  & 
challenge 

Judging  the  credibility  of  a  source 

Making  &  judging  observations 

Making  &  judging  deductions 

Making  &  judging  inductions 

Making  &  judging  value  judgments 

Defining  terms  &  judging  definitions 

Identifying  assumptions 

Deciding  on  an  action 

Interacting  with  others 

Innis,  Evaluating  Critical  Thinking  (Pacific 

1989),  6. 


Table  2.  Critical  Thinking  Dispositions 

Critical  Thinkers  Seek  a  statement  of  the  thesis  or  question 

Seek  reasons 

Try  to  be  well  informed 

Use  credible  sources  and  mention  them 

Take  into  account  the  total  situation 

Keep  their  thinking  relevant  to  the  main 

point 

Keep  in  mind  the  original  or  most  basic 
concern 

Look  for  alternatives 

Are  open  minded  and  seriously  consider 
points  of  view  other  than  their  own;  reason 
from  starting  points  with  which  they 
disagree  without  letting  disagreement 
interfere  with  their  reasoning 
Take  a  position  and  change  a  position  when 
evidence  and  reasons  are  sufficient  to  do  so 
Critical  Thinkers  Seek  as  much  precision  as  the  subject 

permits 

Deal  in  an  orderly  manner  with  parts  of  a 
complex  whole 
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Table  2 — continued 

Employ  their  critical  thinking  abilities 
Are  sensitive  to  the  feehngs,  level  of 
knowledge,  and  degree  of  sophistication  of 
others 

Source:  Stephen  P.  Norris  and  Robert  H.  Ennis,  Evaluating  Critical  Thinking  (Pacific 
Grove,  CA:  Critical  Thinking  Press,  1989),  6. 

Summary 

This  chapter  has  addressed  the  question,  “What  is  critical  thinking?”  In  order  for  us 
to  speak  the  language  of  critical  thinking  there  must  be  a  shared  meaning  of  just  what  this 
term  entails.  Eor  this  paper  critical  thinking  is  the  propensity  and  skills  to  engage  in  an 
activity  with  reflective  skepticism  focused  on  deciding  what  to  reasonably  believe  or  do. 
Three  broad  criteria  can  be  extracted  from  this  definition:  thinking  must  be  reasonable, 
reflective,  and  focused  to  be  considered  critical  thinking.  Critical  thinkers  must  not  only 
possess  the  set  of  skills  we  have  labeled  the  “reason  assessment”  component  (elementary 
clarification,  basic  support,  inference,  advanced  clarification,  and  strategies),  but,  also 
have  the  “critical  spirit”  component.  These  dispositions  rest  fundamentally  on  open- 
mindedness  and  the  desire  to  use  one’s  critical  thinking  abihties  on  one’s  own  and  others’ 
thinking.  As  we  shall  see,  this  definition  is  useful  in  evaluation,  instruction,  and 
curriculum  development.  The  usefulness  comes  from  the  desired  characteristics  which  are 
fisted  in  Tables  1  and  2.  Eor  instance,  these  fists  provide  the  content  specifications  for 
developing  tests  that  evaluate  students’  critical  thinking,  the  next  topic  of  this  paper. 

Notes 

^  J.  Halonen,  “Demystifying  critical  thinking,”  Teaching  of  Psychology  22,  no.  1 
(1995):  75-81. 

^  R.  Ennis,  “Critical  thinking  and  subject  specificity:  Clarification  and  needed 
research,”  Educational  Researcher  18,  no.  3  (1989):  4-10. 

^  John  McPeck,  Critical  Thinking  and  Education  (  New  York:  St  Martins,  1981) 
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Notes 


J.  Halonen,  “Demystifying  critical  thinking,”  Teaching  of  Psychology  22,  no.  1 
(1995):  75-81. 

^  Stephen  P.  Norris  and  Robert  H.  Ennis,  Evaluating  Critical  Thinking  (Pacific 
Grove,  CA:  Critical  Thinking  Press,  1989),  6. 
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Chapter  3 


Measuring  Critical  Thinking 

If  a  thing  exists,  it  exists  in  some  amount.  If  it  exists  in  some  amount,  it 
can  be  measured. 

— E.  L.  Thorndike 

Handbook  in  Research  and  Evaluation 

Evaluation  of  Critical  Thinking  Skills 

The  definition  of  critical  thinking  is  of  great  benefit  in  the  evaluation  process.  It 
enables  us  to  evaluate  the  aggregate,  critical  thinking,  by  looking  at  its  components.  The 
evaluation  can  be  accomphshed  by  a  variety  of  different  methods:  tests,  direct  observation, 
individual  interviews,  student  and  teacher  journals  or  some  combination  of  these.  Tests 
can  be  further  subdivided  into  multi-aspect  tests,  which  are  sometimes  called  general 
critical  thinking  tests  because  they  attempt  to  cover  all  of  the  components  of  critical 
thinking  as  a  whole  and  aspect- specific  tests,  which,  are  only  concerned  with  a  specific 
component  or  part  of  critical  thinking. 

Multiple  choice  tests 

Multiple  choice  tests  are  currently  the  most  commonly  used  assessment  tool  for  a 
variety  of  reasons.  Simphcity,  is  probably  the  best  one.  These  type  of  tests  are  easily 
graded  by  hand  and  can  even  be  machine  scored  making  them  ideal  for  assessing  a  large 
number  of  individuals  on  a  large  number  of  critical  thinking  components.  This  ease  in 
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scoring  allows  for  almost  immediate  feedback  to  the  student  and  timely  reinforcement  of 
proper  behavioral  or  corrective  instruction  if  necessary.  Multiple  choice  tests  also  are 
typically  very  rehable,  that  is  to  say  individual  scores  tend  not  to  vary  much  when  the  test 
is  re-administered.  However,  ensuring  that  the  multiple  choice  test  is  vahd  takes  a  great 
deal  of  expertise  and  definitely  does  not  happen  by  accident.  Additionally,  since  the 
responses  to  multiple  choice  tests  are  simply  to  pick  the  “best  answer”  we  have  no  insight 
into  how  they  arrived  at  their  answer.  Another  problem  is  that  while  this  type  of  test 
works  relatively  well  on  the  “reason  assessment”  component  of  critical  thinking,  for 
instance,  evaluating  inductive  or  deductive  reasoning,  it  is  not  well  suited  for  assessing  the 
“critical  spirit”  disposition  component  such  as  open-mindedness. 

Constructed  response  tests 

Constructed  response  assessments  (essays)  are  not  near  as  easy  to  score  as  multiple 
choice  tests.  The  scoring  process  requires  much  training  and  even  then  test  score 
rehabihty  is  much  less  than  in  multiple  choice  tests.  They  also  take  considerably  more  time 
to  grade  making  the  feedback  process  less  effective  because  of  the  time  lag  between 
student  performance  and  reinforcement  or  corrective  action.  They  do  have  some 
advantages  though.  First  when  developed  properly  they  are  better  suited  for  assessing  the 
“critical  spirit”  critical  thinking  dispositions.  They  also  can  give  us  insight  into  an 
individual’s  thinking  process.  We  not  only  have  their  answer  to  a  question,  but,  we  can 
see  how  they  arrived  at  this  conclusion.  The  Panel  on  Mihtary  Education,  Armed  Services 
Committee,  House  of  Representatives,  101st  Congress,  endorsed  this  method  as  part  of  an 
effort  to  increase  the  rigor  in  PME:  “AU  intermediate  and  senior  level  PME  schools  should 
require  students  to  take  frequent  essay  type  examinations  and  to  write  papers  and  reports 
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that  are  thoroughly  reviewed,  critiqued,  and  graded  by  the  faculty.  Examinations  should 
test  the  student’s  knowledge,  his  abihty  to  think,  and  how  well  he  can  synthesize  and 
articulate  solutions.”^ 

Currently,  this  type  of  test  is  used  at  ACSC,  and  although  critical  thinking  is  not 
exphcitly  assessed  in  the  examinations,  many  of  the  components  of  critical  thinking  impact 
one’s  grade.  Focusing  on  the  question,  clarifying,  using  credible  sources,  and  looking  for 
alternatives  are  just  a  few  of  the  skills  and  dispositions  that  must  be  used  in  ACSC  essay 
exams.  In  this  manner,  the  evaluations  do  seem  to  fulfill  the  Panel  on  Mihtary  Education 
intent,  but  because  of  the  long  time  lag  between  the  evaluation  and  the  feedback  process  a 
valuable  learning  and  reinforcement  opportunity  is  being  missed.  By  the  time  an  exam  is 
returned,  both  the  student  and  instructor  are  both  heavily  involved  in  the  next  course,  thus, 
the  feedback  process  is  superficial  and  of  httle  value.  One  other  point  of  constructive 
criticism  is  that  the  current  in  class  exams  are  heavily  dependent  on  good  typing  skills, 
something  that  may  or  may  not  have  originally  been  intended.  There  are  some 
commercially  available  alternatives  that  could  be  used  at  ACSC  to  exphcitly  evaluate 
critical  thinking. 

Commercial  tests  currently  available  in  both  multiple  choice  and  essay  format  are 
identified  by  Norris  and  Ennis  in  their  book  entitled.  Evaluating  Critical  Thinking}  They 
have  broken  them  out  into  multi-aspect  (a  test  that  attempts  to  measure  ah  or  most  of  the 
critical  thinking  skiUs  and  dispositions)  and  aspect- specific  (a  test  that  concentrates  on  a 
single  skiU  or  disposition  of  critical  thinking)  categories  and  fisted  them  in  alphabetical 
order: 
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Multi-aspect  tests 

1.  Cornell  Critical  Thinking  Test,  Level  X  (1985),  developed  by  Robert  Ennis  and 
Jason  MiUman  and  appropriate  for  students  in  grades  4  through  college.^  There  is  also  a 
Level  Z  version  which  is  appropriate  for  advanced  high  school  students,  college  students 
and  adults.  Both  versions  focus  on  the  evaluative  aspects  of  critical  thinking. 

2.  Ennis-Weir  Critical  Thinking  Essay  Test  (1985),  is  designed  for  secondary  and 
college  students."^  This  test  is  non-machine  scorable  and  is  characterized  as  a  diagnostic 
and  research  tool.  The  authors  indicate  that  the  test  could  be  used  in  an  exploratory 
pretest-posttest  design,  providing  educated  guesses  about  the  effect  of  a  specific 
curriculum. 

3.  New  Jersey  Test  of  Reasoning  Skills  (1983)  by  Virginia  Shipman.  The  test  can  be 
used  with  students  from  fifth  grade  to  college  level.^  About  half  of  the  test  items  are 
concerned  with  deduction,  which  puts  it  somewhere  in  the  middle  between  a  multi-aspect 
test  and  a  aspect- specific  one. 

4.  Watson-Glaser  Critical  Thinking  Appraisal  (1980),  designed  for  students  in  Grade 
9  and  above,  by  Watson  and  Glaser.'’  Two  equivalent  alternate  forms  are  provided.  This 
is  undoubtedly  the  most  popular  of  the  critical  thinking  tests.  Its  vahdity  and  rehabihty 
have  been  extensively  studied  and  confirmed  and  a  wealth  of  statistical  information  is 
available  on  this  measure.^  Sample  questions  and  additional  information  is  included  in 
Appendix  A  of  this  report.^ 
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Aspect-Specific  Tests 

1.  Logical  Reasoning  (1955),  by  Hertzka  and  Guilford,  consists  of  two  parallel  parts 
that  were  standardized  on  two  populations-high  school  students  and  college  students.^ 
The  authors  refer  to  it  as  a  logical  evaluation. 

2.  Test  on  Appraising  Observations  (1983),  by  Norris  and  King,  is  intended  for 
students  ranging  from  junior  high  school  to  college  level.  The  test  items  cover  a  set  of 
principles,  and  items  were  selected  after  students  were  interviewed. 

Although  each  of  these  tests  provide  some  help  in  studying  critical  thinking, 
Sormumen  and  Chahpa  found  that  the  use  of  one  commercial  test  did  not  adequately 
measure  gains  that  were  apparent  when  quahtative  measures  were  apphed.^^  For  instance, 
a  deeper  level  of  discussion  and  greater  enthusiasm  for  learning  expressed  by  students  in 
their  experimental  groups  were  apparent  to  both  teachers  but  were  not  measured. 

Observation 

Another  type  of  assessment  is  observation.  The  observation  can  be  self  observation 
as  in  a  student  journal  or  by  thinking  out  loud  while  working.  It  could  also  be  the 
observations  of  a  teacher  or  researcher.  As  mentioned  this  type  of  quahtative  study  can 
contribute  much  information  when  used  in  conjunction  with  quantitative  tests.  It  aUows 
the  thinking  process  to  be  explored  and  can  provide  insight  into  how  and  why  a  particular 
answer  was  given.  Care  must  be  exercised  to  avoid  the  Hawthorne  Effect,  in  these  types 
of  observations  though.  This  effect  was  first  noted  during  a  study  of  the  relationship 
between  factory  fighting  and  worker  productivity.  Researchers  found  that  as  fight  levels 
went  up  so  did  productivity.  However,  they  also  found  that  the  converse  was  true. 
Lower  fight  levels  in  factories  also  produced  increased  productivity.  The  main  factor  in 
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increased  productivity  was  not  the  light  level,  but  the  fact  that  the  workers  knew  they 
were  being  closely  studied.  Thus  observation  should  be  as  discrete  as  possible  since  the 
act  of  observing  someone  tends  to  effect  their  behavior.  The  Top  Performer  Program  in 
each  course  is  determined  through  an  observation  process  of  students  and  the  instructors. 
Although  it  is  more  of  a  leadership  evaluation  in  general,  intuitively,  it  would  seem  to 
involve  some  critical  thinking  aspects  also. 

Surveys 

StiU  another  type  of  measurement  is  surveys.  One  such  survey  is  the  Motivated 
Strategies  for  Learning  Questionnaire  (MSLQ)  developed  by  Pintrich  &  colleagues. 
This  questionnaire  is  a  self-report  consisting  of  40  motivational  and  65  cognitive  strategy 
questions.  It  is  a  Likert  type  scaled  instrument  and  a  typical  question  would  be:  When  a 
theory  is  presented  in  class,  I  decide  if  there  is  good  supporting  evidence?  An  answer  of  1 
would  indicate  that  this  was  not  true  for  the  individual,  while  an  answer  of  7  indicates  that 
this  is  very  true  of  the  individual.  Table  3  gives  a  complete  look  at  the  variables  of  the 
MSLQ.^^ 

Summary 

This  chapter  has  addressed  the  question,  “How  can  critical  thinking  be  assessed?” 
Using  the  definition  synthesized  in  chapter  one,  it  is  possible  to  assess  critical  thinking  by 
looking  at  its  component  skills  and  dispositions.  The  assessment  can  attempt  to  measure 
only  a  particular  skill  or  disposition  (aspect-specific)  or  a  broad  range  (multi- aspect)  and 
may  come  in  the  form  of  multiple  choice,  essay,  observation,  or  survey.  Each  type  of 
assessment  has  strengths  and  weaknesses  and  the  best  possible  situation  would  entail  the 
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use  of  several  different  types  to  give  a  complementary  assessment  picture.  However,  if 
this  is  not  practical,  the  Watson-Glaser  Critical  Thinking  Test  is  the  most  widely  used 
assessment  and  probably  the  best  single  measurement  available.  These  assessment  tools 
have  been  used  extensively  by  researchers  to  evaluate  critical  thinking  programs  in  order 
to  determine  what  works  in  fostering  critical  thinking  and  what  doesn’t — our  next  topic 
of  discussion. 


Table  3.  Variables  of  the  Motivated  Strategies  for  Learning 


1.  Motivation 

1.1  Values 

1.11  Goal  orientation 

1.12  External  reward  focus 

1.13  Task  value 

1)  usefulness 

2)  value 

3)  interest 

1 .2  Expectations 

1.21  Control  beliefs 

1 )  for  learning 

2)  for  using  strategies  to  learn 

1 .22  Self-efficacy  beliefs 

1)  self-confidence  to  learn 

2)  self-confidence  to  master  (and  perform) 

1.23  Affect 

1)  learning  anxiety 

2)  performance  anxiety 

3)  evaluation  anxiety 

2.  Cognitions  and  Metacognitions 

2.1  Cognitive 

2.11  Rehearsal 

1)  recalling 

2)  repeating 

3)  recognizing 

2.12  Elaboration 

1)  summarizing 

2)  paraphrasing 

2.13  Organization 

1)  outlining 
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2)  integrating 

3)  synthesizing 

2.2  Metacognitive 

2.21  Planning 

2.22  Monitoring 

2.23  Regulating 

2.24  Critical  thinking 

2.3  Strategy  Management 

2.3 1  Use  of  time 

2.32  Study  environment 

2.33  Effort  management 

2.34  Help-seeking  behaviors 

(Abstracted  from  Pintrich  and  Johnson,  1990;  pp.  87-88) 

Notes 

^  House  Committee  on  Armed  Service,  Panel  on  Military  Education,  Report  of  Panel 
on  Military  Education  of  the  One  Hundredth  Congress,  101st  Cong.,  1st  sess.,  1989, 
169-170. 

^  Stephen  P.  Norris  and  Robert  H.  Ennis,  Evaluating  Critical  Thinking  (Pacific 
Grove,  CA:  Critical  Thinking  Press,  1989),  54-97. 

^  Ibid.,  61-68. 

^  Ibid.,  80-83 
^  Ibid.,  73-75. 

^  Ibid.,  57-61. 

^  Richard  Bruce  Modjeski,  “A  Critical  Analysis  and  Evaluation  of  the  Vahdity  and 
Rehabihty  of  the  Cornell  Critical  Thinking  Test  and  the  Watson-Glaser  Critical  Thinking 
Test,”  (Ph.D.  diss..  University  of  Southern  California,  1982),  51-95. 

^  Watson-Glaser  Manual,  Critical  Thinking  Appraisal,  (New  York:  Harcourt,  Brace 
&  World,  Inc.,  1964). 

^  Stephen  P.  Norris  and  Robert  H.  Ennis,  Evaluating  Critical  Thinking  (Pacific 
Grove,  CA:  Critical  Thinking  Press,  1989),  88-89. 

Ibid.,  90-92. 

Carolee  Sormunen,  and  Marilyn  Chalupa,  “Critical  Thinking  Skills  Research: 
Developing  Evaluation  Techniques,”  Journal  of  Education  for  Business,  January/Eebruary 
1994,  174. 

Paul  Pintrich  et  al,  A  manual  for  the  use  of  the  Motivated  Strategies  for  Learning 
Questionnaire  (Ann  Arbor,  Michigan:  National  Center  for  Research  to  Improve  Post- 
Secondary  Teaching  and  Eearning,  The  School  of  Education,  The  University  of  Michigan, 
1991). 
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Chapter  4 


Fostering  Critical  Thinking 


The  boom  in  efforts  to  explicitly  teach  thinking  has  provoked  a  wide 
variety  of  disparate  approaches,  claims  and  advocates.  They  all  tell  us 
what  we  should  be  aiming  at  in  teaching  thinking  and  how  best  to  do  it. 
Often,  however,  hidden  beneath  the  use  of  more  or  less  the  same 
terminology  are  quite  different  goals  and  means. 


—  Robert  J.  Swartz 
Teaching  Thinking:  Issues  &  Approaches 

Instructional  Techniques 

Educators  are  facing  a  tremendous  challenge  developing  and  implementing  programs 
to  foster  critical  thinking.  Pohticians,  mihtary  leaders,  and  educators  aU  agree  that  in 
order  to  compete  in  the  new  world  order  and  exercise  the  rights  and  responsibihties  of 
citizenship  that  Americans  need  to  be  able  to  think  critically.  Although  there  is  a  strong 
agreement  on  critical  thinking  as  an  outcome,  there  is  much  debate  over  what  instructional 
methods  best  bring  it  about.  ^  Four  terms  related  to  instructional  approaches  used  to  foster 
critical  thinking  must  now  be  introduced.  First,  infusion  is  the  process  of  inserting  critical 
thinking  instruction  within  subject-matter  instruction.  The  general  principles  of  critical 
thinking  are  made  exphcit  within  subject-matter  instruction  and  students  are  encouraged  to 
use  them  as  tools  for  better  understanding.  The  second  instructional  approach,  immersion, 
is  similar  to  the  first  in  that  there  is  also  a  thought  provoking  kind  of  subject-matter 
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instruction.  However,  in  this  approach  the  critical  thinking  principles  are  not  made 
explicit.  Proponents  of  the  infusion  approach  include  Glaser  (1984),  and  Resnick  (1987), 
while  McPeck  (1981)  prefers  the  immersion  approach.^  The  third  approach  is  referred  to 
as  the  “general  approach”  and  seeks  to  teach  critical  thinking  separately  from  existing 
subject-matter  instruction.  Non-school  contexts  provide  the  content  about  which  critical 
thinking  is  done.  Examples  of  the  pure  general  approach  are  summarized  by  Kruse  and 
Pressesisen  (1987),  and  Sternburg  and  Bhana  (1986).^  The  final  approach,  termed  the 
“mixed  approach,”  combines  the  general  approach  with  either  infusion  or  immersion. 
Ennis  (1989)  and  Perkins  and  Salomon  (1989)  favor  this  method."^ 

These  different  instructional  approaches  are  the  result  of  a  methodology  debate 
centered  on  whether  critical  thinking  is  generahzable  or  domain  specific.  The  answer  to 
this  question  has  a  dramatic  impact  on  the  instructional  methods  employed.  In  fact,  one 
could  rephrase  the  question  to:  should  critical  thinking  be  taught  as  a  single  body  of 
general,  transferable  skills  or  as  diverse  sets  of  skills,  each  pecuhar  to  an  academic  subject 
area?  If  critical  thinking  is  domain  specific,  the  answer  is  the  latter,  if  critical  thinking  is 
totally  generahzable,  the  answer  is  the  former.  This  question  is  of  vital  importance 
because  it  has  a  direct  impact  on  the  effectiveness  of  the  educational  intervention,  and  so 
we  now  will  attempt  to  answer  it. 

The  Debate:  To  Generalize  or  Not  to  Generalize? 

In  one  of  his  papers  entitled,  “A  Concept  of  Critical  Thinking,”  Ennis  (1962) 
identified  twelve  general  aspects  of  critical  thinking  that  he  claims  are  both  teachable  and 
transferable.^  This  initial  hst  has  been  added  to  and  has  evolved  into  the  hsts  of 
“dispositions”  and  “abihties”  previously  shown  in  Tables  1  &  2.  One  of  the  basic  premises 
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behind  such  inventories  is  that  identifying  the  components  of  critical  thinking  constitutes  a 
major  step  towards  teaching  people  to  become  critical  thinkers.  Once  learned  these 
aspects  of  critical  thinking  can  be  transferred  to  any  other  domain,  given  adequate 
knowledge  of  the  subject  area  in  question. 

McPeck  challenges  the  domain  general  premise  by  claiming  that  since  all  thinking  is 
necessarily  thinking  about  “X,”  there  can  be  no  such  thing  as  a  general  set  of  critical 
thinking  skills  that  can  be  apphed  in  all  contexts  and  that  to  teach  critical  thinking  in 
general  is  fruitless.  As  mentioned  earher,  McPeck  defines  critical  thinking  as  “  the 
appropriate  use  of  reflective  skepticism,”  to  estabhsh  “good  reason  for  various  behefs.” 
He  maintains  that  since  what  constitutes  “good  reasons”  depends  on  the  particular 
epistemological  (ways  of  thinking)  and  logical  norms  of  the  subject  area  in  question, 
critical  thinking  must  vary  from  one  domain  to  the  next.  Rather  than  trying  to  teach 
critical  thinking  in  general,  McPeck  beheves  that  we  should  concentrate  on  giving  students 
a  more  thorough  grounding  in  the  epistemological  underpinning  of  the  key  subject  areas. 

These  views  represent  the  two  extremes  of  the  present  generahzabihty  debate.  This 
dichotomy  in  opinion  has  been  present  for  many  years,  with  each  side  rising  and  faUing  in 
favor  as  new  studies  were  conducted  to  support  their  claims.  Perkins  and  Salomon  (1989) 
give  a  detailed  historical  sketch  of  how  the  trend  has  shifted  from  the  generahst  to  the 
speciahst  and  now  is  moving  back  in  favor  of  the  generahst  camp.'’  Moreover,  using 
theoretical  and  empirical  evidence  from  numerous  studies,  they  suggest  that  both  camps 
may  have  oversimphfied  the  interaction  between  general  strategic  knowledge  and 
specialized  domain  knowledge. 
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Theoretical  Evidence 


Almost  forty  years  ago  heuristics,  or  general  strategies  for  solving  problems,  were 
thought  to  be  the  key  to  excellence  in  thinking.  IQ  and  “g”  for  general  inteUigence  were 
thought  to  represent  some  general  abihties  that  were  able  to  be  used  across  a  broad 
spectrum  of  domains.  Initial  work  in  computer  programming  and  artificial  inteUigence 
also  supported  this  idea.  Means-end  analysis  was  a  very  popular  strategy  that  reUed  on 
these  ideas.  The  study  of  expertise,  most  notably  that  of  grand  master  chess  players,  also 
seemed  to  confirm  that  aU  that  was  needed  was  general  reasoning  abihties  appUed  to  a  few 
simple  rules.  However,  as  the  games  of  grand  master  chess  players  were  analyzed  more 
closely,  it  was  found  that  their  expert  play  was  the  result  of  an  enormous  amount  of 
knowledge  regarding  important  chess  patterns  (domain  specific  knowledge),  a  tremendous 
blow  to  the  idea  that  excellence  in  thinking  was  a  result  of  only  general  abilities. 

Additional  evidence  against  the  generaUzable  theory  also  came  from  research  on 
transfer.  If  exceUence  in  thinking  was  truly  generaUzable,  then  learning  strategies  in  one 
subject  area  or  domain  should  improve  thinking  or  transfer  to  other  subjects  or  domains. 
However,  many  studies  have  failed  to  substantiate  this  anticipated  transfer  (for  a  summary, 
see  Pressley,  Snyder,  and  Carigila-BuU,  1987).’  Computer  programmers  also  found  out 
that  programs  that  were  written  containing  a  vast  amount  of  domain  specific  knowledge 
(strong  method)  were  must  better  at  solving  problems  than  those  programmed  only  with 
general  strategies  or  heuristics  (weak  methods).  Evidence  seemed  overwhelming  in  favor 
of  domain  specific  thinking  skiUs.  General  heuristics  appeared  to  be  no  match  for  a  rich 
domain  specific  database,  stored  in  memory,  accessed  by  the  recognition  process,  and 
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ready  to  be  brought  to  bear  on  the  most  demanding  of  problems.  In  spite  of  this  evidence 
there  was  still  much  that  could  not  be  explained  by  using  this  theory  alone. 

These  difficulties  can  be  overcome  by  abandoning  the  idea  that  general  and  domain 
specific  abihties  are  mutually  exclusive  and  challenging  the  supposed  dichotomy.  Perkins 
and  Salomon  (1989)  suggest  that  “there  are  general  cognitive  skills;  but  they  always 
function  in  contextuahzed  ways.”^  General  strategies  and  domain  specific  knowledge  are 
seen  to  work  hand-in-hand  and  this  requires  special  emphasis  during  instruction  to 
improve  thinking.  New  methods  such  as  Reciprocal  Teaching,  which  will  be  talked  about 
more  later  in  this  paper,  are  producing  dramatic  gains  in  reading,  science  and  other 
subjects.  The  key  seems  to  be  that  general  principles  of  reasoning  need  to  be  taught 
together  with  self- monitoring  practices  and  potential  apphcations  in  varied  contexts. 
Using  this  approach  produces  the  desired  transfer  that  was  previously  so  elusive 
(Nickerson,  Perkins,  &  Smith,  1985).^  Blatz  (1989)  further  clarifies  the  matter  of  transfer 
by  looking  at  dimensions  and  extent  of  transfer  rather  than  simply  transfer  or  non- 
transfer.^'’  He  outlines  four  levels  of  generahty,  with  level  one  being  the  broadest  and 
without  quahfication  to  level  four  at  which  we  must  confront  not  only  universal  logic, 
subject-independent  and  subject- specific  constraints,  but  also  the  particular  case,  problem 
or  issue  to  which  we  are  applying  our  reasoning. 

Perkins  and  Salomon  have  also  proposed  two  mechanisms  by  which  the  transfer  of 
skills  between  domains  takes  place."  The  first,  called  the  “low  road”  to  transfer  depends 
on  extensive  and  varied  practice  of  a  skill  to  near  automaticity.  By  practicing  the  skill  in  a 
large  variety  of  instances  it  is  thought  that  this  skill  will  also  be  apphed  to  similar  instances 
through  the  process  of  stimulus  generahzation.  The  second  mechanism  called  the  “high 
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road”  depends  on  the  learners’  deliberate  mindful  abstraction  of  a  principle.  One  obstacle 
to  this  type  of  transfer  is  that  it  is  often  necessary  to  decontextuahze  a  skill  in  one  domain 
before  it  is  seen  to  apply  to  another  similar  situation  in  another  domain.  This  would  seem 
to  put  the  negative  research  findings  on  transfer  in  another  hght.  Rather  than  implying 
that  transfer  can’t  occur  because  of  domain  specificity,  the  negative  results  reflect  that 
certain  conditions  must  be  met  to  facilitate  transfer. 

The  notion  of  standard  conditions  as  used  in  the  sciences  is  also  put  forth  by  Norris 
(1985)  as  an  explanation  to  the  generahzabihty  debate.  He  asserts  that  it  is  reasonable  to 
maintain  that  given  our  lack  of  detailed  knowledge  concerning  reasoning,  that  the  inabihty 
to  find  reasoning  abihties  that  cut  across  subjects  and  contexts  is  due  only  to  our  inabihty 
to  identify  and  institute  the  required  set  of  standard  conditions.  Ennis  (1989)  adds 
comphmentary  evidence  by  pointing  out  three  versions  of  subject  specificity,  domain, 
epistemological,  and  conceptual.  While  he  concludes  that  conceptual  subject  specificity 
(McPeck’s  view)  has  no  basis  and  is  too  vague  the  other  two  offer  important  insights. 
They  both  emphasize  the  importance  of  background  knowledge.  Epistemological  subject 
specificity  notes  that  there  are  differences  in  what  constitutes  a  good  reason  between  fields 
of  knowledge.  While  domain  specificity  sees  the  importance  of  teaching  for  transfer 
combined  with  frequent  application  of  principles  in  many  different  areas. 

BreU  (1990)  also  took  a  look  at  the  conceptual  battle  between  Ennis  and  McPeck  and 
came  up  with  conclusions  similar  to  Norris.  Namely,  that  although  the  mastery  of 
domain- specific  knowledge  is  the  major  challenge  in  learning  to  think  critically,  as  McPeck 
predicts,  general  concepts  and  strategies  of  thinking  do  exist  and  are  teachable.  In 
addition,  thinking  can  neither  occur  or  be  taught  independently  of  the  epistemological 
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norms  of  some  frame  of  reference  or  knowledge  domain.  Finally,  that  although  both 
general  and  subject-specific  knowledge  are  important  conditions  of  critical  thinking,  the 
teaching  of  neither  adequately  addresses  the  fundamental  problem  of  getting  students  to 
transfer  their  knowledge  and  skills  to  new  areas.  Niaz  (1995)  came  to  a  like  conclusion 
after  looking  at  helping  science  educators  choose  between  domain- specific  and  domain- 
general  strategies  for  enhancing  the  thinking  skills  of  their  students.  He  concluded  that 
the  content-process  dichotomy  is  an  artificial  one  and  in  fact  the  two  approaches  to 
teaching  science  complement  each  other. 

Empirical  Evidence 

Now  let  us  turn  to  the  empirical  evidence  that  has  a  direct  bearing  on  the 
generahzabihty  question.  Sternberg  and  Bhana  (1986)  make  some  general  observations 
about  studies  that  are  associated  with  thinking  programs.^'’  Their  advice  is  to  look  at  the 
data  and  experimental  procedures  used  closely  since  much  of  the  research  and  pubhcation 
is  done  by  individuals  with  possible  conflicts  of  interest.  This  having  been  said,  the  first 
evidence  to  be  looked  at  documents  the  deficiencies  the  majority  of  students  in  this 
country  demonstrate,  de  Sanchez’s  (1995)  study  indicates  that  most  college  students  are 
at  Piaget’s  concrete  thinking  level  rather  than  the  formal  level  ascribed  to  adults.^’  She 
attributes  these  deficiencies,  at  least  in  part,  to  academic  settings  that  emphasize 
memorization  of  isolated  knowledge,  which  are  devoid  of  meaning,  lack  transferabihty, 
and  are  easily  forgotten.  She  characterizes  this  status  quo  as  “incidental  thinking”  and 
asserts  that  for  thinking  skills  to  be  fully  developed  and  transferable  the  thinking  .skill 
instruction  must  be  dehberate.  Her  dehberate  instruction  involves  a  critical  thinking  model 
composed  of  processes,  information,  and  products.  Nine  processes  are  hsted  that  are 
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thought  essential  to  critical  thought:  observation,  comparison,  relations,  classifications, 
ordering,  hierarchical  classifications,  analysis,  synthesis,  and  evaluation.  Several  studies 
(Herrnstein,  Nickerson,  de  Sanchez  &  Swet,  1986;  Reif,  1981)  have  concluded  that  these 
process-based  skills  do  transfer  when  the  instruction  is  designed  to  facihtate  such 
transfer.^* 

Halpern  (1993)  assessed  the  effectiveness  of  critical  thinking  instruction  and 
concluded  that  at  least  seven  quahtatively  different  forms  of  outcome  evaluations  show 
that  thinking  can  be  improved  through  dehberate  instruction  to  do  so.^^  He  cites 
numerous  studies  that  have  included  student  self-reports,  gains  in  IQ  scores,  cognitive 
growth  and  development,  expert-hke  mental  representations,  improvement  in  cognitive 
skills,  and  spontaneous  transfer  that  support  his  claim.  Hanley  (1995)  has  reported 
successful  thinking  instruction  that  focused  on  metacognitive  skills  and  problem  solving.^'’ 
He  beheves  that  in  order  to  be  a  critical  thinker  one  must  first  learn  to  decide  when  a 
specific  cognitive  skill  is  relevant  (a  metacognitive  process)  and  then  successfully  apply  the 
cognitive  skills  to  solve  problems.  This  would  seem  particularly  relevant  as  a  desirable 
outcome  of  ACSC.  Two  more  studies  by  PiroUi  and  Recker  (1994)  also  suggest  that  the 
acquisition  of  cognitive  skills  is  facihtated  by  high  degrees  of  metacognition  (a 
generahzable  skill). Improved  learning  in  the  domain  of  programming  was  related  to 
reflection  on  problem  solutions  that  focused  on  understanding  the  abstractions  underlying 
programming  problems. 

In  a  review  of  more  than  100  studies  on  training  of  learning  strategies,  Belmont, 
Butterfield,  and  Ferretti  (1982)  identified  further  evidence  from  seven  studies  that 
produced  transfer  effects  on  the  cognitive  functioning  of  young  and  mentally  retarded 
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children.^^  Of  these  seven  successful  studies,  six  showed  that  significant  transfer  occurred 
only  when  there  was  instruction  of  self-management  skills,  such  as  goal  setting,  strategy 
planning,  and  self-monitoring — in  addition  to  training  in  specific  skills  involved  in  transfer. 
Another  study  conducted  by  Riesenmy,  Mitchell  and  Hudgins  (1991)  involved  training  38 
children  in  four  thinking  roles. The  roles  were  called  task  definer,  strategist,  monitor, 
and  challenger.  This  experimental  group  showed  superior  retention  scores  on  three 
variables  over  the  control  group.  The  variables  were  use  of  self-directed  thinking  skills, 
amount  of  information  used  in  solutions  and  the  quality  of  their  answers. 

Another  instructional  procedure  that  numerous  research  studies  (for  a  summary  see 
Rosenshine  and  Meister,  1994)  have  deemed  successful  is  called  Reciprocal  Teaching.^"^  It 
has  two  major  features,  the  first  is  instruction  and  practice  in  four  comprehension- 
fostering  strategies:  question  generation,  summarization,  prediction,  and  clarification.  The 
second  consists  of  using  dialogue  as  a  vehicle  for  learning  and  practicing  these  four 
strategies.  The  teacher  begins  by  modehng  the  strategies  and  then  gradually  the  students 
become  more  active  until  finally  the  teacher  is  merely  a  facihtator  of  the  process.  One 
final  example  of  the  successful  use  of  the  infusion  approach  is  given  by  Zohar, 
Weinberger,  and  Tamir  (1994).^^  Their  carefully  designed  instruction  incorporated 
activities  for  developing  specific  critical  thinking  skills  that  were  incorporated  into  biology 
curriculum.  Improved  critical  thinking  skills  were  observed  both  in  the  biology  context 
and  non-biology  everyday  topics,  further  suggesting  generalization  across  domains. 

Summary 

Modern  attempts  to  foster  critical  thinking  have  been  only  moderately  successful.  A 
consistent  theme  is  that  a  short-term  course  of  study  covering  one  subject  matter  area 
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does  not  have  enduring  effects  on  solving  problems  in  other  subject  matter  domains. 
However,  this  doesn’t  necessarily  mean  that  critical  thinking  skills  are  domain  specific. 
The  research,  both  the  theoretical  and  empirical  data,  suggests  that  the  answer  to  the 
original  question  (Is  critical  thinking  generahzable  or  domain  specific?),  is  that  it  is  neither. 
The  question  assumes  a  dichotomy  and  research  points  to  a  dynamic  almost  synergistic 
effect  between  the  two. 

The  general  theme  of  agreement  in  the  hterature  is  that  efforts  need  to  be  focused  on 
bringing  together  context- specific  knowledge  with  general  “critical  thinking”  skills 
instruction  with  dehberate  efforts  made  to  facihtate  transfer  to  real  life  apphcations. 
McPeck  is  one  of  the  few  exceptions  that  stiU  clings  to  supporting  a  strictly  immersion 
(imphcit)  approach  to  teaching  critical  thinking.  Although  his  position  has  forced 
considerable  theoretical  considerations  to  be  evaluated  further,  the  majority  of  evidence 
points  to  an  infusion  (exphcit)  approach  to  fostering  critical  thinking  as  having  the  greatest 
probability  of  success. 
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Chapter  5 


Conclusions 


Preparing  airman  for  future  rather  than  past  wars  involves  constant 
encouragement  of  open-minded  thinking  to  ensure  intellectual  growth.  Yet 
American  PME  schools,  including  those  of  the  Air  Force,  have  often 
suffered  from  rigidity,  normalization,  and  a  lack  of  dynamism  in  nurturing 
thought. 
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Research  Implications 


Curriculum 

The  current  curriculum  at  ACSC  (see  table  4  for  a  summary)  focuses  on  ten  courses 
of  study  and  a  research  program  comprising  approximately  600  academic  contact  hours/ 
From  the  course  descriptions  in  the  Air  University  catalog  it  is  clear  that  critical  thinking  is 
required  throughout  the  curriculum.  However,  actual  exphcit  instruction  on  the  skills  and 
dispositions  that  make  up  critical  thinking  is  limited.  The  only  mention  of  critical  thinking 
instruction  is  in  the  War  Theory  course.  Here  students  are  introduced  to  critical  thinking 
via  the  process  of  Clausewitzian  critical  analysis.  This  analysis  consists  of  three  steps: 
estabhsh  the  facts,  trace  effects  back  to  root  causes,  and  consider  and  evaluate  alternate 
courses  of  action.  Notice  that  these  three  steps  depend  heavily  on  the  mastery  of  the  five 
critical  thinking  “reason  assessment”  skills  hsted  in  Table  1.  These  necessary  skills  are 
only  implicitly  addressed  in  this  course. 
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While  critical  thinking  is  imphcitly  addressed,  through  the  immersion  approach,  the 
research  outhned  in  chapter  four  of  this  paper  suggests  that  a  more  exphcit  approach 
would  be  beneficial.  Critical  thinking  instruction  is  more  successful  when  efforts  are 
focused  on  bringing  together  context- specific  knowledge  and  general  critical  thinking 
skills  instruction  with  dehberate  efforts  made  to  facihtate  transfer  to  real  hfe  apphcations. 
This  would  point  to  an  infusion  or  mixed  approach  to  the  curriculum,  that  exphcitly 
teaches  critical  thinking  skills  and  dispositions  as  part  of  each  of  the  ten  courses  currently 
offered,  as  having  the  greatest  probability  of  fostering  critical  thinking. 

This  is  not  to  say  the  curriculum  is  bad.  The  current  curriculum  provides  a  sohd 
foundation  of  domain  specific  knowledge  in  airpower,  joint  operations,  leadership  and 
thinking  skills  within  these  domains.  The  overwhelming  majority  (90.84%)  of  students 
perceive  the  ACSC  experience  as  increasing  their  abihty  to  think  critically  when 
conducting  an  in-depth  analysis  of  complex  situations  or  problems.^  Moreover,  ACSC 
graduates  perception  of  the  curriculum  has  risen  over  the  last  several  years  from  merely 
satisfactory  to  approaching  outstanding  on  a  five  point  scale. ^  The  point  is  that  research 
documented  in  this  paper  suggests  the  inclusion  of  exphcit  critical  thinking  skiUs  and 
disposition  instruction  would  result  in  an  additional  quantitative  improvement  in  students’ 
critical  thinking  abilities. 


Table  4.  ACSC  Curriculum  Summary 


Course 

Approximate  %  of  Curriculum 

War  and  Conflict 

3% 

War  Theory 

10% 
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Table  4 — continued 

Course  Approximate  %  of  Curriculum 

Strategic  Environment  14% 

Operational  Structures  16% 

War  and  Conflict  Resolution  3% 

Joint  Operations  and  Campaign  Concepts  1 1  % 

Air  Power  and  Campaign  Planning  22% 

Joint  Warrior  9% 

Leadership  and  Command  9% 

Force  2025  3% 

Research  N/A 

Source:  Air  Command  and  Staff  College,  Air  University  Catalog,  1995-96,  n.p.;  on-line, 
Internet,  6  February  1997,  available  from  http://www.au.af.mil/au/cat/acsc.html. 

Instructional  Techniques 

Over  the  last  three  years,  ACSC  educators  have  made  a  major  paradigm  shift  in  the 
way  students  are  instructed.  Passive  learning  through  lectures  and  teacher  directed 
instruction  in  rote  memorization  of  isolated  facts  is  no  longer  the  way  they  do  business. 
Seminar  based  active  learning  is  facihtated  with  an  integrated  curriculum,  ample  student 
discussion,  case  studies  and  computer  simulations.  In  fact,  these  active  learning  activities 
account  for  approximately  three-fourths  of  the  resident  curriculum."^  Nonetheless,  new 
methods  of  instruction  should  be  explored.  For  example,  reciprocal  teaching,  which  first 
teaches  students  learning  and  thinking  strategies  and  then  moves  on  to  more  student 
directed  learning  is  strongly  endorsed  by  this  research.  Again,  this  is  consistent  with  the 
infusion  or  mixed  approach  to  fostering  critical  thinking. 
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Evaluations 


No  approach  should  be  followed  blindly  though,  “measures  of  success”  are  just  as 
important  in  education  as  they  are  in  campaign  planning.  Looking  specifically  at  the 
current  evaluation  system  in  place  at  ACSC,  it  is  unclear  what  these  “measures  of  success” 
for  critical  thinking  are.  The  ACSC  goal  of  fostering  critical  thinking  is  vague  and  the 
current  evaluation  methods  at  ACSC  provide  decision  makers  with  only  information  that  is 
largely  descriptive  and  anecdotal  in  nature.^  As  mentioned  earher,  the  current  evaluation 
system  (essay  exams,  direct  observation,  and  surveys)  can  provide  some  quahtative 
information,  but  these  methods  take  a  considerable  amount  of  time  and  effort  to  overcome 
their  inherent  vahdity  and  rehabihty  problems.  This  is  why  a  vahd  and  rehable  quantitative 
measurement  of  critical  thinking,  that  is  easily  administered  and  graded,  is  needed. 

Administering  an  internationally  recognized  measurement  of  critical  thinking,  such  as 
the  Watson-Glaser  Appraisal,  at  the  beginning  of  ACSC  and  at  the  end  would  have 
tremendous  benefits.  First,  it  would  provide  a  quantifiable  measurement  (measure  of 
success)  upon  which  the  critical  thinking  goal  of  ACSC  could  be  evaluated.  This  data 
could  also  be  correlated  to  the  present  grading  system  to  estabhsh  credibihty  and  vahdate 
the  program.  For  instance,  as  mentioned  earher  over  90%  of  in  residence  ACSC  students 
beheve  that  the  current  program  has  increased  their  abihty  to  think  criticaUy,  this 
additional  data  could  be  used  to  evaluate  these  perceptions  and  quantify  the  value  added 
by  the  program.  FinaUy,  such  a  measurement  would  estabhsh  a  very  valuable  database. 
Benchmarks  could  be  estabhshed  and  used  from  year  to  year  to  evaluate  the  effects  of  any 
changes  in  the  ACSC  program.  Many  other  studies,  such  as  correlations  between  Myer- 
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Briggs  Personality  Type  Indicator  and  critical  thinking  gain  could  also  add  to  the  body  of 
knowledge. 

Staff  Development 

There  is  a  definite  fink  between  instructors’  critical  thinking  abihty  and  the  progress  of 
their  students  in  this  area.'’  By  administering  the  critical  thinking  measurement  discussed 
above  to  instructors,  staff  development  instruction  in  critical  thinking  could  be  provided 
on  an  as  needed  basis.  Across  the  board  in  education,  low  instructor  abihty  in  critical 
thinking  is  a  primary  cause  in  students’  low  scores  in  this  area.^  It  would  also  be 
interesting  to  see  the  change  in  an  instructor’s  critical  thinking  score  over  the  term  of  a 
three  year  assignment  at  ACSC. 


Critical  Analysis 

It  is  now  time  to  tie  aU  the  theory  just  presented  to  the  reahty  at  hand  and  provide 
some  final  conclusions  and  recommendations.  This  research  project  sought  a  framework 
that  could  be  used  to  assess  the  ACSC  program  to  foster  critical  thinking  and  now  it  is 
only  appropriate  to  use  the  Clausewitzian  critical  analysis  to  do  just  that.  The  first  part  of 
the  analysis  is  to  estabhsh  the  facts.  We  did  this  by  synthesizing  a  definition  for  critical 
thinking,  the  propensity  and  skills  to  engage  in  an  activity  with  reflective  skepticism 
focused  on  deciding  what  to  reasonably  believe  or  do.  Next,  the  facts  revolving  around 
how  to  assess  critical  thinking  were  sorted  through.  From  this,  the  Watson-Glaser  Critical 
Thinking  Appraisal  was  identified  as  having  the  best  overall  characteristics  of  vahdity, 
reliability,  and  ease  of  use. 
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Tracing  effects  back  to  root  causes  is  step  two  of  the  critical  analysis.  Here,  the 
analysis  starts  to  run  into  some  problems.  The  data  ACSC  has  compiled  on  the  “effects” 
of  its  critical  thinking  program  are  largely  descriptive  and  thus  very  hard  to  trace  to  any 
root  causes.  In  general,  we  might  be  able  to  attribute  perceived  student  gains  in  critical 
thinking  to  an  active  versus  a  passive  curriculum,  but,  without  quantitative  data  and 
additional  studies  nothing  can  be  determined  for  certain.  The  bottom  hne  would  seem  to 
be  that  the  current  program  does  foster  critical  thinking,  however,  with  what  effectiveness 
and  efficiency?  The  Watson-Glaser  appraisal  could  prove  most  helpful  in  answering  these 
questions.  These  limitations  also  carry  over  into  the  final  area  of  analysis,  evaluating  and 
considering  alternative  courses  of  action. 

Chapter  four  evaluated  the  research  concerning  alternative  teaching  methods  to  foster 
critical  thinking  Although  this  was  the  longest  and  most  theoretical  part  of  this  paper,  the 
conclusion  was  finally  drawn  that  ACSC  could  better  foster  critical  thinking  through  more 
exphcit  critical  thinking  skills  instruction,  an  infusion  versus  the  current  immersion 
approach.  Again,  better  quantitative  critical  thinking  data  would  be  required  to  prove  this 
in  practice.  Although,  this  analysis  was  limited  somewhat  by  a  current  lack  of  data,  it 
provides  a  helpful  framework  and  suggests  ways  to  improve  upon  the  current  critical 
thinking  program. 

Recommendations 

The  following  recommendations  are  provided  based  on  the  discussions  above: 

1.  A  critical  thinking  measurement,  such  as  the  Watson-Claser  Critical  Thinking 
Appraisal  should  be  administered  to  incoming  students  and  instructors  and  again 
upon  ACSC  completion. 

2.  Data  from  these  appraisals  should  be  used  to  set  measurable  goals  and  monitor  the 
effects  of  program  changes. 
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3.  Critical  thinking  should  be  addressed  more  exphcitly  in  the  ACSC  curriculum  using 
an  infusion  approach. 

4.  Data  should  also  be  correlated  against  other  databases,  in  particular  the  Myer- 
Briggs  Type  Indicators  to  explore  relationships  between  student  inputs  (i.e. 
personahty  type,  gender,  age,  etc.),  the  school  environment  (i.e.  curriculum, 
teaching  methodologies,  etc.),  and  outcomes  (i.e.  TPs,  grades,  etc.). 

The  potential  impact  of  improving  ACSC’s  critical  thinking  program  is  immense. 
Innovative  problem  solving,  critical  analytical  thought  and  sound  professional  judgment 
have  been  and  will  continue  to  be  keys  in  our  mihtary  leadership  achieving  battlespace 
dominance.  Thus,  any  improvements  that  can  be  gained  by  critical  analysis  of  the  program 
will  provide  our  future  leaders  with  a  great  advantage.  This  research  and  the 
accompanying  recommendations  are  only  a  potential  starting  point,  but,  one  upon  which  a 
firm  foundation  of  continuous  improvement  can  be  built. 


Notes 

^  Air  Command  and  Staff  College,  Air  University  Catalog,  1995-96,  n.p.;  on-line, 
Internet,  6  February  1997,  available  from  http://www.au.af.mil/au/cat/acsc.html. 

^.Air  Command  and  Staff  College,  AY  96  End-of-Year  Survey  Results,  n.p.  On-line. 
Internet,  18  March  1997.  Available  from  http://spock.au.mil/bbs/Academics/EOY96- 
l.ppt. 

^Ibid. 

"^Air  Command  and  Staff  College,  Self  Study  Report,  March  1997,  p.  3-12.  On-line. 
Internet,  18  March  1997.  Available  from  http://spock.au.mil/bbs/PAJE/ascs_bbs.doc. 

^Brenda  E.  Roth,  “Student  Outcomes  Assessment  of  Air  Command  and  Staff  College: 
An  Evaluative  Study”  (Ph.D.  diss..  University  of  Virginia,  1996),  93. 

^Robert  J.  Swartz  and  D.  N.  Perkins,  Teaching  Thinking:  Issues  and  Approaches 
(Pacific  Grove,  CA:  Critical  Thinking  Press,  1990),  194. 

^  R.  Paul,  Critical  Thinking:  How  to  Prepare  Students  for  a  Rapidly  Changing  World 
(Sonoma  CA:  Eoundation  for  Critical  Thinking,  1995),  47-90. 
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Appendix  A 


Watson-Glaser  Critical  Thinking  Appraisal 

The  Watson-Glaser  Critical  Thinking  Appraisal  is  composed  of  a  series  of  five  tests 
which  require  the  apphcation  of  important  “reason  assessment”  abihties  of  critical 
thinking.  The  five  sub-tests  cover  the  following  areas:  Inference,  recognition  of 
assumptions,  deduction,  interpretation,  and  evaluation  of  arguments. 

The  “Inference  Test”  gauges  ones’  abihty  to  discern  the  degree  of  truth  or  falsity  of 
inferences  drawn  from  the  data  that  you  are  given.  The  abihty  to  recognize  unstated 
assumptions  in  statements  is  sampled  in  the  “Recognition  of  Assumptions  Test.”  The 
“Deduction  Test”  samples  the  abihty  to  reason  properly  when  given  a  statement  of 
premise;  to  recognize  the  relation  of  imphcation  between  propositions.  The  abihty  to 
weigh  evidence  and  make  vahd  generahzations  is  sample  in  the  “Interpretation  Test.” 
Finahy,  the  “Evaluation  of  Arguments  Test”  samples  the  abihty  to  distinguish  strong  and 
relevant  arguments  from  ones  which  are  weak  or  irrelevant.  The  80-item  appraisal  can  be 
completed  in  50  minutes,  and  produces  a  single  score  that  can  be  compared  with  a  wide 
range  norm  groups.  The  materials  to  administer  a  pre  and  post  test  at  ACSC  would  cost 
approximately  $2,000. 

Two  examples  of  typical  questions  are  given  below: 
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Interpretation  Example:  A  study  of  vocabulary  growth  in  children  from  eight  months  to 
six  years  shows  that  the  size  of  spoken  vocabulary  increases  from  zero  words  at  age  eight 
months  to  2562  words  at  age  six. 

1.  None  of  the  children  in  this  study  had  learned  to  talk  by  the  age  of  six  months. 
Does  this  conclusion  foUow  or  not?  (This  conclusion  follows  beyond  a  reasonable  doubt 
since,  according  to  the  statement,  the  size  of  the  vocabulary  at  eight  months  was  zero 
words) 

2.  Vocabulary  growth  is  the  slowest  during  the  period  when  children  are  learning  to 
walk.  Does  this  conclusion  follow  or  not?  (This  conclusion  does  not  follow  since  there  is 
no  information  given  that  relates  growth  of  vocabulary  to  walking.) 

Deduction  Example:  Some  holidays  are  rainy.  All  rainy  days  are  boring.  Therefore — 

1.  No  clear  days  are  boring.  (The  conclusion  does  not  follow.  You  cannot  teU  from 
the  statements  whether  or  not  clear  days  are  boring.  Some  may  be.) 

2.  Some  hohdays  are  boring.  (The  conclusion  necessarily  follows  from  the  statement 
since,  according  to  them,  the  rainy  holidays  must  be  boring.) 

3.  Some  hohdays  are  not  boring.  (The  conclusion  does  not  follow,  even  though  you 
may  know  some  holidays  are  very  pleasant.) 
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